In meta-analyses, we can apply techniques to partially mitigate the risk of distortions caused by publication and reporting bias. Some of these strategies focus on improving the study search process, while others involve statistical adjustments.
Study Search - Several specific approaches can help minimize the risk of bias:
Include Grey Literature: such as dissertations, preprints, government reports, and conference proceedings;
Backward and Forward Snowballing: Review the references of included studies (backward snowballing) and identify papers that have cited those studies (forward snowballing).
Search in Multiple Databases
Comprehensive Search Strategies
Contacting Study Authors for Unpublished Data
Statistical methods: Statistical methods can help detect signs of publication bias by analyzing patterns in the data, such as asymmetry in funnel plots or a correlation between study size and effect size. While these methods cannot directly confirm publication bias, they highlight its possible presence. Some techniques also adjust for potential bias, providing corrected estimates of the overall effect size to improve the reliability of the meta-analysis
The reasoning is straightforward: Publication bias exists because SIGNIFICANT RESULTS are more likely to be published. Since LARGER studies have a HIGHER probability of producing significant results due to their larger sample sizes, publication bias disproportionately affects SMALL STUDIES, leading to an overrepresentation of large effect sizes in the published literature.
Studies with small sample sizes ➞ Low precision (high standard errors) ➞ Wider confidence intervals ➞ Greater likelihood of non-significant results.
This is precisely why methods for assessing publication bias focus on the issue of small-study effects. As the name suggests, these approaches are particularly concerned with the unique challenges posed by small studies.
It is common practice to assess small-study effects using funnel plots. A funnel plot is a scatter plot that displays:
Take a look at the example below:
In the absence of small-study effects (no publication bias), the studies should approximately follow the idealized funnel shape displayed in the plot. Is this true in our example? Subjectively, we can say that the studies do follow the shape reasonably well.
Studies located in the top part of the plot have low SE and should lie closely together, and not far away from the pooled effect size.
Studies located in the lower part of the plot, with increasing SE, the funnel “opens up”, and effect sizes are expected to scatter more heavily to the left and right of the pooled effect.
In this example there are 2 small studies with very low effect sizes to “balance out” the one with very high effects.
Now, let’s explore an example of funnel plot asymmetry:
In this case, the studies deviate from the idealized symmetrical funnel shape, suggesting the presence of small-study effects or potential publication bias.
There are three small studies with very high effect sizes located in the bottom-right corner of the plot (‘Shapiro’, ‘Kang’, and ‘Danitz-Orsillo’).
However, there are no corresponding small studies in the bottom-left corner of the plot with low or negative effect sizes to “balance out” these high-effect studies.
This imbalance contributes to the asymmetry in the funnel plot and may indicate the presence of small-study effects or publication bias.
Another concerning observation is that the study with the greatest precision in our sample, conducted by de ‘Vibe’, does not align well with the expected funnel pattern. Its effect size is noticeably smaller than anticipated for a study with such high precision.
Overall, the dataset exhibits an asymmetrical pattern in the funnel plot, which could suggest publication bias. It is possible that the three small studies with high effect sizes represent outliers that happened to achieve significant results, while a hidden pool of unpublished studies with similar standard errors but smaller, non-significant effects remains unaccounted for. This imbalance highlights the potential for selective reporting to skew the overall findings.
A good way to inspect how asymmetry patterns relate to statistical significance is to generate contour-enhanced funnel plots (Peterset al., 2008). Such plots can help to distinguish publication bias from other forms of asymmetry.
Usually, the statistical significance thresholds are 0.95 and 0.99, which equals p < 0.05 and p < 0.01, respectively. We see that the funnel plot now contains three shaded regions. We are particularly interested in the p ≥ 0.05, p < 0.05, and p < 0.01 regions.
In the funnel plot, the standard error of the treatment estimates is plotted on the y-axis by default which is likely to be the best choice (Example A). However, we also can plot on the y-axis the 1/SE (precision), as demonstrated in Example B.
Studies appear to be MISSING in areas of statistical NON-SIGNIFICANT or SIGNIFICANT NEGATIVE/LOWER effect (flattened in orange) ➤ Increase the possibility of that the asymmetry is due to publication bias.
See the example below:
Studies appear to be MISSING in areas of HIGHER STATISTICAL SIGNIFICANCE (flattened in pink) ➤ Reduces the plausibility that publication bias.
See the example below:
Look after other sources of ‘begin’ funnel plot asymmetry:
Between-study heterogeneity;
Study procedures were different in small studies;
High risk-of-bias - Low quality studies;
Random (by chance).
We will now make this more concrete by generating a funnel plot our
selves. In the meta package, the funnel()
function can be used to print a funnel plot for a meta-analysis object.
Here, we produce a funnel plot for our metabin()
meta-analysis object.
library(meta)
library(readxl)
## Author Year event.e n.e event.c n.c p2y12_type
## 1 Almendro-Delia et al. 2015 188 3973 116 2076 Clopidogrel
## 2 ATLANTIC Trial 2014 30 906 19 952 Ticagrelor
## 3 Danchin et al. 2018 9 360 18 360 Clopidogrel
## 4 Load&Go Trial 2013 2 56 0 56 Clopidogrel
## 5 CIPAMI Trial 2012 1 164 4 171 Clopidogrel
## 6 Fabris et al. 2022 17 237 17 300 Clopidogrel
## 7 Postma et al. 2014 46 1345 35 1096 Clopidogrel
## 8 Dorler et al. 2011 55 1635 110 1076 Clopidogrel
## 9 Lupi et al. 2016 5 143 13 143 Ticagrelor
## 10 Redfors et al. 2019 1960 37840 528 6964 Clopidogrel
## 11 Yudi et al. 2018 25 892 96 1915 Clopidogrel
## 12 Alexander et al. 2008 12 1029 55 2756 Ticagrelor
## 13 Fefer et al. 2009 12 217 6 166 Clopidogrel
## 14 Lev et al. 2008 1 165 2 127 Clopidogrel
## overal_risk_of_bias pretreatment _timing %_HTN
## 1 Serious-risk of Bias Pre-hospital 30.0
## 2 Low-risk of Bias Pre-hospital 28.8
## 3 Low-risk of Bias Pre-hospital 25.9
## 4 Moderate-risk of Bias Pre-hospital 52.3
## 5 Low-risk of Bias Pre-hospital 49.8
## 6 Moderate-risk of Bias Pre-hospital 30.5
## 7 Serious-risk of Bias Pre-hospital 20.3
## 8 Moderate-risk of Bias Pre-hospital 40.1
## 9 Serious-risk of Bias Pre-hospital 60.6
## 10 Moderate-risk of Bias Pre-hospital 40.5
## 11 Moderate-risk of Bias Pre-hospital 43.4
## 12 Serious-risk of Bias In-hospital 61.4
## 13 Serious-risk of Bias In-hospital 32.6
## 14 Serious-risk of Bias In-hospital 70.1
m.acm <- metabin(event.e, n.e, event.c, n.c,
data = ma$acm,
method = "MH",
method.tau = "DL",
sm = "RR",
studlab = Author)
summary(m.acm)
## RR 95%-CI %W(common) %W(random)
## Almendro-Delia et al. 0.8469 [0.6761; 1.0608] 11.0 13.0
## ATLANTIC Trial 1.6591 [0.9407; 2.9262] 1.3 8.3
## Danchin et al. 0.5000 [0.2277; 1.0981] 1.3 5.9
## Load&Go Trial 5.0000 [0.2455; 101.8321] 0.0 0.6
## CIPAMI Trial 0.2607 [0.0294; 2.3078] 0.3 1.2
## Fabris et al. 1.2658 [0.6606; 2.4255] 1.1 7.3
## Postma et al. 1.0710 [0.6951; 1.6502] 2.8 10.1
## Dorler et al. 0.3291 [0.2403; 0.4506] 9.6 11.8
## Lupi et al. 0.3846 [0.1408; 1.0508] 0.9 4.3
## Redfors et al. 0.6832 [0.6227; 0.7495] 64.4 14.2
## Yudi et al. 0.5591 [0.3626; 0.8619] 4.4 10.1
## Alexander et al. 0.5844 [0.3142; 1.0867] 2.2 7.6
## Fefer et al. 1.5300 [0.5864; 3.9914] 0.5 4.6
## Lev et al. 0.3848 [0.0353; 4.1970] 0.2 1.0
##
## Number of studies: k = 14
## Number of observations: o = 67120 (o.e = 48962, o.c = 18158)
## Number of events: e = 3382
##
## RR 95%-CI z p-value
## Common effect model 0.6887 [0.6380; 0.7433] -9.57 < 0.0001
## Random effects model 0.7274 [0.5673; 0.9327] -2.51 0.0121
##
## Quantifying heterogeneity:
## tau^2 = 0.1108; tau = 0.3328; I^2 = 73.7% [55.4%; 84.5%]; H = 1.95 [1.50; 2.54]
##
## Test of heterogeneity:
## Q d.f. p-value
## 49.46 13 < 0.0001
##
## Details on meta-analytical method:
## - Mantel-Haenszel method (common effect model)
## - Inverse variance method (random effects model)
## - DerSimonian-Laird estimator for tau^2
## - Mantel-Haenszel estimator used in calculation of Q and tau^2 (like RevMan 5)
## - Continuity correction of 0.5 in studies with zero cell frequencies
funnel(m.acm,
studlab = TRUE,
bg = "lightblue",
cex = 2,
cex.studlab = 1,
random = T,
fixed = F)
title("Funnel Plot for All-Cause Mortality")
# Define fill colors for contour
funnel(m.acm,
yaxis = "invse",
ylab = "Precision (1/SE)",
studlab = TRUE,
bg = "lightblue",
cex = 2,
cex.studlab = 1,
random = T,
fixed = F,
contour = c(0.95, 0.99))
legend(x = 5.2, y = 21,
legend = c("p ≥ 0.05", "p < 0.05", "p < 0.01"),
fill = c("white" , "#808080", "#E6E6E6"))
title("Example A\nContour-Enhanced Funnel Plot for All-Cause Mortality")
metabias(m.acm, method.bias = "Egger")
## Linear regression test of funnel plot asymmetry
##
## Test result: t = 0.34, df = 12, p-value = 0.7363
## Bias estimate: 0.2498 (SE = 0.7246)
##
## Details:
## - multiplicative residual heterogeneity variance (tau^2 = 4.0812)
## - predictor: standard error
## - weight: inverse variance
## - reference: Egger et al. (1997), BMJ
metabias(m.acm, method.bias = "Peters")
## Linear regression test of funnel plot asymmetry
##
## Test result: t = 0.33, df = 12, p-value = 0.7446
## Bias estimate: 50.8053 (SE = 152.3709)
##
## Details:
## - multiplicative residual heterogeneity variance (tau^2 = 16.5424)
## - predictor: inverse of total sample size
## - weight: inverse variance of average event probability
## - reference: Peters et al. (2006), JAMA
funnel(trimfill(m.acm))